Introduction: Accurate clinical trial matching remains a major barrier to oncology trial enrollment, particularly due to the difficulty of extracting structured eligibility criteria from unstructured text. While multiple commercial and open-source efforts (e.g., MatchMiner, TrialGPT) have aimed to improve matching via manual structuring or large language model (LLM)-based search enhancements, they often fall short in capturing the complex constraints present in real-world trial eligibility criteria. To address this, we developed, which employs a novel LLM-based Attribute-Criteria Extraction (ACE) engine to convert free-text trial eligibility criteria into true structured criteria across a comprehensive set of domain-specific attributes. This abstract presents the first large-scale accuracy analysis of ACE in the domain of follicular lymphoma (FL) clinical trials.

Methods: ACE relies on attribute-specific prompts curated in a Prompt Workbench, enabling biomedical subject matter experts (SMEs) to refine extraction logic iteratively. It handles both discrete (e.g. hemoglobin, LDH) and compound (e.g. renal function) attributes, including named OR logic constructs (e.g., MeetsGELF) and complex therapy-type mappings (e.g. identifying therapies by class or constituent). Importantly, ACE converts extracted logic into conjunctive normal form (a set of top level AND criteria with OR criteria and potential criteria negations pushed to lower levels) to facilitate efficient and interpretable patient-trial matching via EXACT's matching engine.

Results: We evaluated ACE performance using the most recent 100 FL trials from ClinicalTrials.gov (all posted from October 2020 to May 2025), manually labeled by SMEs for 84 core patient attributes and 533 associated attribute criteria. These include diagnostic, lab, genetic, prior therapy, and functional status parameters. Precision, recall, and F1 (harmonic mean of precision and recall) were computed using micro-averaged metrics across all attributes, ensuring performance reflects real-world frequency distributions.

Overall, ACE achieved a precision (how often inferred attribute is correct) of 80%, recall (how often attributes referred to in criteria are successfully extracted) of 84%, and F1 score of 81.8% across all extracted criteria. Performance varied by attribute category: numeric lab values (e.g., ANC minimum or bilirubin maximum) were extracted with 98% precision, while treatment history criteria involving therapy type requirements showed lower precision (~77%), largely due to ambiguities in free-text descriptions and complexity of parsing treatment classes and exclusions. Accuracy did not vary significantly across phases or modalities, except that complex requirements regarding previous treatment had lower precision. These differences guide targeted prompt refinement efforts. For comparison, other LLM-based trial matchers use frequency that “doctor labeled gold standard trial” for patient vignette are in top 5 ranked trials to assess accuracy. EXACT achieves 94% (compared to 87.3% for TrialGPT) with this method. Note however, that this metric does not actually assess true precision eligibility as our primary metrics do.

Prompts are dynamically constructed using variable placeholders (e.g. $therapy type, $mutation_status) to generalize across specific values without redundant engineering for each attribute.

These findings support the feasibility of EXACT for structured extraction for trial eligibility logic automatically, reliably and at scale. The ACE engine's performance in FL trials demonstrates high accuracy can be achieved without manual rule writing or per-trial prompt editing, despite ambiguous, inconsistent and complex clinical language. Conclusions and Future Directions: With accurate ACE extraction of attribute criteria, patients and doctors can view clearly expressed and exact eligibility criteria for the most complex clinical trials and quickly determine whether they are eligible. Future work will expand ACE evaluation to other cancers, implementing error-type classification, and refining prompt logic to further improve therapy-related criteria performance. With these report results and planned enhancements, EXACT holds promise as a foundational infrastructure for real-time patient trial matching in precision oncology. The EXACT software is made available freely to patient support organizations and cancer foundations.

This content is only available as a PDF.
2025
Sign in via your Institution